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FOREWORD 


The  United  States  Army  is  concerned  with  continuous  self-development  of  all  soldiers. 
Self-development  often  begins  with  an  accurate  assessment  of  strengths  and  weaknesses.  This 
literature  review  was  conducted  to  examine  the  potential  of  multirater  assessments,  also  known  as 
360-degree  evaluations,  for  systematic  use  by  the  Army.  Possible  utilization  of  multirater 
assessments  includes  applications  which  would  promote  accurate  self-assessment,  encourage 
developmental  growth  in  areas  relevant  to  leadership,  and  reinforce  organizational  values.  A 
growing  literature  on  the  use  of  multirater  assessments  in  commercial  organizations  is  reviewed 
and  discussed  in  this  report. 

This  review  was  completed  by  the  ARI  Research  Unit  at  Fort  Leavenworth  in  conjunction 
with  the  Leadership  Research  and  Assessment  Division  of  the  Center  for  Army  Leadership 
(CAL),  and  the  results  were  briefed  to  the  Director,  CAL,  February  1997. 


ZITAM.  SIMUTIS 
Technical  Director 
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MULTIRATER  ASSESSMENT  PROCESS  -  A  LITERATURE  REVIEW 


EXECUTIVE  SUMMARY 


Research  Requirement: 

To  provide  background  information  for  the  multirater  survey  project  promulgated  by  the 
Deputy  Chief  of  Staff  for  Personnel. 

Procedure: 

Major  databases  were  searched  for  articles  and  reports  on  the  use  and  effectiveness  of 
multirater  feedback  evaluation  systems.  These  reports  and  articles  were  reviewed  and  analyzed, 
and  major  findings  and  issues  are  reported. 

Findings: 

A  review  of  the  current  literature  on  multirater  feedback  evaluations  indicates  that  this 
process  could  increase  self-awareness,  thereby  encouraging  self-development.  However,  more 
empirical  data  are  necessary  to  conclude  that  multirater  feedback  has  a  lasting  and  beneficial 
impact  on  behavior  and  performance. 

Utilization  of  Findings: 

The  literature  review  will  be  used  by  ARI,  the  Center  for  Army  Leadership,  and  other 
organizations  exploring  the  possible  utility  of  multirater  assessment  as  a  tool  for  self-development 
within  the  United  States  Army. 
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MULTIRATER  ASSESSMENT  PROCESS  -  A  LITERATURE  REVIEW 


Introduction 

The  United  States  Army  faces  dramatic  organizational  challenges  brought  about  by  global 
political  changes,  increased  operating  tempo,  constrained  resources  and  undefined  missions 
(Homer,  1995).  Similar  challenges  in  the  corporate  world  have  emphasized  the  need  for  quick 
and  efficient  adaptation,  more  teamwork,  more  accountability,  and  improved  performance  at  less 
cost.  In  response  to  quickly  changing  needs,  many  organizations  are  turning  to  an  employee 
evaluation  and  feedback  process  which  helps  to  direct  employee  professional  development  while 
promoting  the  organizational  culture  most  conducive  to  meeting  organizational  needs.  This 
evaluation  process  is  the  multirater  feedback  process,  which  involves  receiving  performance 
feedback  from  multiple  perspectives  (i.e.,  from  subordinates,  peers,  self,  supervisors  and  possibly 
customers). 

The  purpose  of  this  paper  is  to  report  research  findings  relevant  to  the  multirater  feedback 
concept  and  to  discuss  the  implications  of  this  body  of  research  for  professional  development  of 
Army  leaders.  The  evolution  of  the  multirater  process  will  be  reviewed.  The  multirater  method 
will  be  compared  to  more  traditional  methods  of  assessment,  and  benefits  and  objectives  will  be 
discussed.  Finally,  recommendations  and  caveats  for  implementation  within  the  Army 
professional  development  system  will  be  discussed. 

Evolution  of  the  Multirater  Feedback  Concept 

The  multirater  feedback  process,  sometimes  referred  to  as  a  360-degree  assessment,  is  rooted 
in  both  the  developmental  and  the  performance  appraisal  fields.  The  use  of  performance 
appraisals  began  in  the  late  1800’s  as  production  processes  became  mechanized  during  the 
Industrial  Age.  At  first,  supervisors  used  trait  checklists  for  employee  performance  appraisals. 
These  eventually  evolved  into  various  scales  and  behavioral  descriptions  which  were  designed  to 
improve  the  quality  of  such  appraisals.  During  the  I960’ s  and  70’s  the  most  prevalent  form  of 
performance  appraisal  was  a  comparison  of  individual  results  with  organizational  goals  and 
strategies  (i.e.,  management  by  objectives).  The  use  of  supervisor  feedback  for  the  explicit 
purpose  of  employee  personal  and  career  development  did  not  become  popular  until  the  late 
1980’s.  Up  until  this  point,  supervisor  feedback  was  mainly  perceived  as  evaluative,  not 
developmental  (Edwards  &  Ewen,  1996). 

The  total  quality  management  (TQM)  initiative  is  most  commonly  credited  with  originating  the 
use  of  multirater  feedback  for  developmental  appraisals.  This  initiative  took  organizational 
surveys  and  extended  them  into  division  level  surveys,  then  department  level  and  finally  service 
representative  level,  (i.e.,  customer  service)  surveys.  Now  multirater  feedback  from  subordinates, 
peers,  self,  superiors  and  customers  is  used  by  organizations  that  have  become  concerned  with 
career  development,  diversity  management,  fair  reward  decisions,  accurate  and  valid  performance 
measures,  and  legal  protection  for  decisions  relating  to  personnel. 
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Generally,  one  is  evaluated  on  the  particular  competencies  which  are  highly  valued  within  his 
or  her  organization  and  which  impact  individual  performance  (e.g.  communication  skills, 
creativity).  One  potential  outcome  of  a  multirater  evaluation  is  that  organizational  values,  which 
guide  behavior  and  decision-making  at  all  levels,  can  be  disseminated  and  reinforced  throughout 
the  organization.  A  second  and  equally  important  potential  consequence  of  this  process  is  that 
individual  employees  can  gain  an  increased  self-awareness  of  the  competencies  related  to 
organizational  values.  With  an  increased  self-awareness,  employees  are  empowered  to  develop  the 
skills  and  competencies  that  are  important  to  the  organization,  and  which  help  to  guarantee  a 
successful  career  track. 

Currently,  multirater  feedback  systems  are  an  amalgamation  of  several  traditional  feedback 
sources.  Thus,  it  is  instructive  to  examine  the  relevant  empirical  literature  regarding  those  sources. 
The  following  section  reviews  supervisor  appraisal  feedback,  peer  review  feedback,  upward 
feedback  research  and  self-assessment  feedback  . 

Traditional  Feedback  Methods 
Supervisor  Appraisal. 

Research  has  shown  that  supervisor-only  feedback  tends  to  be  inflated,  time-consuming,  and 
does  not  differentiate  levels  of  performance  nor  motivate  employees  to  improve  (Edwards  & 
Ewen,  1996).  Supervisor  appraisals  and  multirater  feedback  share  common  problems,  such  as 
questionable  validity  and  rating  bias  (e.g.,  response  consistency,  leniency,  halo  and  stereotyping) 
(Borman,  1974).  However,  supervisor  assessments  have  additional  problems,  including 
favoritism,  politics,  and  varying  degrees  of  care  when  making  assessments.  No  one  enjoys  being 
the  bearer  of  bad  news  and  that  includes  supervisors,  who  sometimes  skirt  issues  related  to 
employee  weaknesses  and  poor  performance.  Generally,  supervisors  avoid  doing  evaluations  for 
as  long  as  they  can,  and  eventually  provide  only  flat,  non-specific  information  (Edwards  &  Ewen, 
1996). 

Also,  supervisors  may  have  insufficient  opportunity  or  motivation  to  observe  employee 
performance.  Schneier  &  Beatty  (1978)  found  that  supervisors  may  draw  inferences  about  one’s 
ability  to  lead  or  manage  based  on  the  productivity  of  that  person’s  work  unit,  as  opposed  to 
actually  observing  that  person’s  individual  behavior.  While  productivity  of  the  work  unit  indicates 
to  the  supervisor  what  is  getting  done,  it  does  not  indicate  how  things  are  getting  done.  How 
things  are  being  done  may,  or  may  not,  mesh  with  the  vision  and  values  of  the  organization  as  a 
whole. 

Traditional  performance  appraisal  and  the  multirater  feedback  process  differ  in  two  important 
ways.  First,  performance  appraisals  are  conducted  primarily  for  evaluation  and  have 
organizational  consequences  (e.g.,  pay  raises,  promotions,  or  transfers).  A  multirater  assessment 
is  usually  conducted  to  provide  guidance  for  employee  development  and  performance 
improvement.  Secondly,  supervisory  ratings  provide  only  one  source  of  evaluation  data,  whereas 
multirater  feedback  provides  data  from  multiple  sources.  Therefore,  it  is  unlikely  that  a  single 
person  can  have  undue  influence  in  the  feedback  the  employee  receives.  Multirater  feedback 
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recognizes  the  complexity  of  management  and  the  value  of  input  from  different  sources  (Becker  & 
Klimoski,  1989).  Additionally,  multiple  ratings  allow  testing  of  consistency  of  leader  behaviors 
and  reliability  of  the  information  gathered. 

Peer  Appraisal. 

Lateral  feedback  has  been  shown  to  be  reliable  (peers  are  consistent  with  one  another),  valid 
(highest  rated  people  eventually  receive  the  promotions),  and  perceived  as  highly  credible  to  those 
who  receive  the  feedback.  Nonetheless,  in  smaller  organizations,  there  may  be  a  shortage  of  peers 
with  sufficient  contact  to  provide  quality  feedback.  Generally,  while  peers  tend  to  provide  honest 
feedback  (especially  when  the  feedback  is  for  developmental  purposes  only)  there  is  always  the 
fear  that  a  peer  may  provide  false  negative  information  in  an  attempt  to  derail  a  target’s  career,  or 
to  promote  their  own  career.  While  feedback  from  a  single  peer  has  its  shortcomings,  input  from 
multiple  sources  makes  it  easier  to  spot  false  negative  feedback. 

Upward  Feedback. 

Research  indicates  that  upward  feedback  (subordinate  evaluations)  is  a  better  predictor  for 
performance  improvement  than  self  and  supervisor  assessment  (Wilson,  O’Hare  &  Shipper, 

1990).  Subordinate  assessments  provide  important  details  on  how  supervisors  achieve  results, 
and  on  the  reasons  why  communication  failures  occur  (Berman  &  Hellweg,  1988).  Subordinate 
assessments  have  also  been  associated  with  improved  teacher  performances  (Tuckman  &  Oliver, 
1968),  and  with  eliciting  modest  changes  in  managerial  behavior  (Hegarty,  1974;  Van  Velsor, 
Ruderman,  and  Philips,  1991).  Subordinates  are  in  an  excellent  position  to  view  and  evaluate 
leadership  behaviors.  Although  subordinates  may  provide  information  of  slightly  higher  quality 
than  supervisors  or  peers,  organizational  cultures  with  a  highly  structured  hierarchical  system  may 
find  the  use  of  subordinate  reports  incompatible  with  organizational  values.  Additionally,  there 
may  be  fears  that  subordinates  may  “gang  up”  against  a  particular  ratee,  or  that  the  ratee  may 
retaliate  against  a  source  of  negative  feedback.  Therefore,  anonymity  and  other  safeguards  are 
essential  (Edwards  &  Ewen,  1996). 

According  to  Bemardin,  Dahmus  &  Redmon  (1993),  managers  and  supervisors  have  several 
concerns  regarding  upward  (subordinate)  feedback.  Such  concerns  include:  1)  supervisors  may 
focus  on  pleasing  subordinates  in  an  effort  to  get  higher  appraisals,  2)  the  authority  of  the 
manager  could  be  undermined  by  the  pressure  of  upward  appraisal  and  the  implications  of  low 
evaluations  for  the  manager’s  status  within  the  organization,  3)  subordinates  lack  the  ability, 
aptitude,  training  or  necessary  job  information  to  provide  valid  ratings,  4)  subordinates  may  be 
reluctant  to  be  candid  about  their  bosses  for  fear  of  repercussions,  or  they  may  inflate  ratings  in 
order  to  gain  points  with  the  manager,  5)  employees  who  are  being  pushed  the  hardest  by  their 
supervisors  may  rate  those  supervisors  more  harshly,  6)  managers  may  also  be  uncertain  about 
how  to  interpret  subordinate  appraisal  relative  to  ratings  from  other  sources ,  such  as  their  boss. 
Most  of  these  concerns  can  be  dealt  with  by  instituting  safeguards  that  ensure  confidentiality,  and 
with  employee  training  in  giving  and  receiving  feedback. 
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Self  Assessment. 


In  order  to  succeed  in  an  organization,  individuals  must  continuously  self-assess  their 
performance,  skills  and  potential  (Ashford,  1989).  Several  studies  indicate  the  self  to  be  the  most 
available  and  trustworthy  source  of  feedback  (Greller  &  Herold,  1975;  Hanser  &  Muchinsky, 
1978).  Simply  participating  in  a  self-evaluation  makes  one  increasingly  self-aware,  more 
accepting  of  feedback  from  others,  and  more  committed  to  goal  setting  (Ashford,  1989). 
Additionally,  self-ratings  can  promote  personal  development,  improve  communication  between 
supervisors  and  subordinates,  and  clarify  differences  of  opinion  between  supervisors  and  other 
managers  (Cummings  &  Schwab,  1973). 

Carver  &  Scheier  (1981)  suggest  that  control  theory  provides  the  framework  for 
understanding  the  self-regulation  processes.  This  theory  proposes  that  individuals  survive  by  a 
continual  process  of  matching  their  behavior  to  a  goal  or  standard.  Goals  are  obtained  when 
individuals  accomplish  the  following  tasks:  1)  setting  standards  for  their  behavior,  2)  detecting 
discrepancies  between  their  behavior  and  those  standards  (self-assessment),  and  3)  enacting 
behaviors  to  reduce  these  discrepancies. 

Ashford  (1989)  adds  two  self-assessment  tasks  to  this  list  for  the  individual  within  an 
organization.  Specifically,  individuals  must  be  able  to  assess  whether  the  standards  they  have 
chosen  to  guide  their  behavior  actually  enable  goal  attainment.  Secondly,  individuals  must 
develop  proficiency  in  seeing  and  assessing  their  behavior  in  a  manner  consistent  with  how  others 
perceive  and  evaluate  it.  Effective  self-regulation  in  organizations  requires  attention  to  issues  of 
validity  of  one’s  self-assessment  with  respect  to  the  assessments  of  others  (Ashford,  1989). 

Another  perspective,  taken  by  symbolic  interactionists,  argues  the  relevance  of  others’ 
influence  in  the  assessment  of  self  (Cooley,  1902;  Mead,  1934).  The  basic  tenet  of  their  theory  is 
that  one  develops  a  self-concept  and  makes  self-assessments  based  on  his/her  beliefs  about  how 
others  perceive  and  evaluate  them.  Accordingly,  one  is  better  able  to  predict  and  control  social 
interactions  if  they  are  able  to  see  themselves  as  others  see  them  (Mead,  1934). 

Understanding  and  anticipating  others’  assessments  is  crucial  for  an  individual  working  within 
an  organization.  Given  that  only  the  individual  is  fully  aware  of  any  situational  constraints  on 
his/her  behavior,  self-assessment  could  be  more  accurate  than  the  subjective  ratings  of  others,  but 
it  is  the  other’s  assessments  which  have  important  consequences  for  the  individual.  It  should  be 
noted  that  most  research  regards  other’s  ratings  as  the  objective  criterion  (i.e.,  one  is  an  accurate 
self-rater  only  if  his/her  ratings  match  the  ratings  given  by  others). 

Most  research  on  self-assessment  indicates  that  compared  to  ratings  by  others,  self- 
assessments  are  often  inflated,  unreliable,  invalid,  biased  and  inaccurate  (Yammarino  & 

Atwater,  1993).  Van  Velsor,  Ruderman,  and  Young  (1991)  reported  that  only  10%  of  managers 
studied  saw  themselves  as  others  saw  them.  They  also  found  that  overrating  self  seems  to  be  the 
most  common  profile.  Prior  research  indicates  that  regardless  of  the  dimension  being  measured  or 
the  sources  of  other  ratings,  the  average  self-ratings  tend  to  be  more  favorable  than  other’s  ratings 
(  Ashford,  1989;  Harris  &  Schaubroeck,  1988;  Landy  &  Farr,  1980;  Mabe  &  West,  1982; 
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Thornton,  1980;  and  Wohlers  &  London,  1989).  Furthermore,  a  literature  review  conducted  by 
Mabe  and  West  (1982)  indicates  that  the  average  correlation  between  self-ratings  and  the 
accuracy  criteria  was  small.  Additionally,  Shrauger  and  Schoeneman  (1979)  reviewed  50  studies 
on  the  congruencies  between  self  and  others’  evaluations  and  found  that  about  one-half  of  the 
studies  showed  no  correlation  at  all,  and  the  majority  of  the  studies  showed  either  significant  but 
low  correlations  or  ambiguous  results.  Additionally,  their  research  showed  modest  to  strong 
correlations  between  individuals’  perceptions  of  themselves  and  the  way  they  assumed  others  saw 
them.  Finally,  a  significant  correlation  was  found  between  individuals’  views  of  how  others  saw 
them  and  how  others  actually  saw  them. 

Why  does  self-assessment  inaccuracy  occur? 

According  to  Ashford  (1989)  self  -assessment  inaccuracy  may  occur  because  of  three  common 
difficulties  individuals  experience  during  the  self-assessment  process.  First  individuals  must  be 
able  to  obtain  the  information  necessary  to  assess  their  performance  accurately  and  they  must 
interpret  this  information  honestly.  Additionally,  they  must  balance  the  desire  to  understand  what 
they  should  be  doing  and  how  well  they  have  done  with  the  costs  to  the  ego  that  they  suffer  by 
hearing  negative  feedback.  Furthermore,  individuals  must  obtain  standard  feedback  information 
within  a  social  environment  in  which  information  seeking  may  have  particularly  negative  symbolic 
meaning  as  a  sign  of  weakness  or  insecurity. 

Accurately  rating  one’s  self  is  not  an  easy  task.  However,  supplementing  self-assessments 
with  feedback  from  others  should  increase  self-awareness  of  strengths  and  weaknesses. 

Consistent  with  the  symbolic  interactionist  perspective,  research  by  Van  Velsor  et.  al.  (1991), 
indicates  that  one  month  after  attending  a  management  development  program  and  receiving  initial 
feedback,  about  80%  of  managers  modified  their  self-assessment  in  the  expected  direction  on  one 
or  more  scales  after  feedback  from  others. 

Implications  of  inaccurate  self-assessments 

At  least  two  studies  (Bass  &  Yammarino,  1991;  and  Atwater  &  Yammarino,  1992)  indicate 
that  inaccurate  self-raters  tended  to  be  poorer  performers  than  people  who  rated  themselves  as 
others  rated  them.  According  to  Bass  and  Yammarino  (1991),  individuals  with  inaccurate  self- 
assessments  will  misdiagnose  their  strengths  and  weaknesses.  Both  inflated  and  deflated  self- 
assessments  can  adversely  affect  job-relevant  decisions.  Inflated  self-assessments  are  associated 
with  career  derailment  (McCall  &  Lombardo,  1983).  Research  indicates  that  derailment  may 
result  because  those  with  inflated  self-assessments  have  self-aspirations  and  expectations  which 
exceed  others’  perceptions  of  their  capabilities.  Conversely,  those  who  under-estimate  their 
abilities  and  skills  will  tend  to  set  a  low  aspiration  level  and  will  underachieve  (Bandura,  1982). 
According  to  Dweck  and  Leggett  (1988),  the  effects  of  deflated  self-assessment  may  be  mediated 
by  how  individuals  view  a  task  -  as  either  a  performance  goal  or  as  a  learning  goal.  Individuals 
with  performance  goals  seek  favorable  judgments  of  their  competence;  whereas,  individuals  with 
learning  goals  seek  to  increase  their  competence.  Individuals  with  low  self-assessments  and 
learning  goals  are  more  likely  to  put  effort  into  a  task  than  the  individuals  with  low  self- 
assessment  and  performance  goals. 
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Another  variable  possibly  mediating  the  relationship  between  deflated  self-assessment  and 
performance  is  task  understanding.  Underestimation  of  one’s  understanding  of  the  task  may  lead 
to  excessive  information-seeking,  which  takes  away  from  the  time  to  complete  the  task. 
Overestimation  of  task  understanding  may  lead  to  a  failure  to  perform  the  task  correctly. 

Implications  of  accurate  self-assessments 

Accurate  self-assessment  is  associated  with  numerous  benefits  for  the  individual  and  the 
organization.  For  instance,  Bandura  (1977),  proposed  that  self-assessment  of  aptitude  and  skills 
influence  an  individual’s  decisions  about  how  to  allocate  his/her  efforts,  how  intensely  to  work, 
and  how  long  to  persist  in  the  face  of  challenges. 

Individuals  with  accurate  self-assessments  are  in  a  better  position  to  take  corrective  actions, 
such  as  increasing  effort  or  changing  task  strategies  when  needed.  If  individuals  believe  they  are 
doing  fine,  when  in  fact,  they  are  not,  they  are  unable  to  make  the  necessary  corrections  to  their 
behavior  (Nilsen  &  Campbell,  1993). 

McCauley  and  Lombardo  (1990)  found  that  self-awareness  was  positively  related  to  superiors’ 
assessments  of  promotability  -  the  more  accurate  the  self-assessment,  the  more  likely  an  individual 
would  be  to  be  promoted.  According  to  Bass  and  Yammarino  (1991)  naval  officers  with  more 
accurate  self-perception  attained  higher  ranks  and  were  rated  as  more  promotable  by  their 
superiors 

A  process  like  three  hundred  sixty  degree  assessment,  which  is  thought  to  enhance  self- 
assessment  accuracy,  offers  an  improved  alternative.  Although  any  of  the  traditional  feedback 
methods  have  associated  benefits,  most  people  find  the  combined  feedback  from  their  supervisors, 
peers,  and  subordinates  highly  motivating.  Individuals  can  discount  feedback  from  a  supervisor 
(or  any  one  source),  but  hearing  the  same  constructive  criticism  from  peers  and  subordinates  with 
whom  he/she  has  personally  worked  is  more  likely  to  motivate  improvement. 

Benefits  and  Objectives  of  the  Multirater  Feedback  Process 

A  properly  implemented  multirater  assessment  enhances  employee  development  and  provides 
an  organizational  culture  that  defines  itself  as:  more  empowering,  team  oriented,  productive, 
participative,  and  as  offering  equal  opportunity  for  it’s  members  to  succeed  and  advance. 
Researchers  have  documented  the  numerous  advantages  of  multiple  rater  systems,  including  the 
following:  enhanced  ability  to  observe  and  measure  various  job  facets  (Borman,  1974),  greater 
reliability,  fairness,  and  ratee  acceptance  (Latham  &  Wexley,  1994),  enhanced  communications 
and  performance  ( London,  Wohlers  &  Gallagher,  1990;  Bemardin  &  Beatty,  1987),  more  input 
for  merit  evaluation  and  compensation  adjustment  (McEvoy  &  Buller,  1 987)  and  improved 
defensibility  of  the  performance  appraisal  program  from  a  legal  standpoint  (Bemardin  &  Beatty, 
1984). 
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For  most  organizations,  the  objectives  of  the  multirater  process  include: 

•  developing  an  individual  to  his/her  full  potential 

•  aligning  individual  and  team  behavior  with  organizational  vision  and  values 

•  focusing  on  competency  based  rewards 

•  providing  fair  and  accurate  performance  measures 

•  supporting  a  commitment  to  continued  learning 

•  reinforcing  and  communicating  other  organizational  initiatives 

•  stimulating  an  organization’s  employees  to  improve  particular  skills  and  essential 
competencies 

•  enhancing  employees’  awareness  of  strengths  and  developmental  need 

Features  of  an  Effective  multirater  Instrument  and  Process 

The  advantages  of  a  multirater  assessment  process  will 
issues  are  addressed.  The  following  section  is  a 
discussion  of  those  issues.  Note  that  the  particular 
emphasis  during  development  and  implementation 
may  vary  somewhat  depending  on  the  organization’s 
stated  objectives  .  The  use  of  360  assessments  as 
part  of  an  employee  evaluation  process,  for  example, 
will  require  particular  attention  to  a  match  between 
assessment  items  and  job  performance  standards. 

However,  the  following  concerns  must  all  be 
addressed  to  some  extent  in  any  multirater 
assessment  implementation. 

1.  A  relevant  assessment  instrument:  developing  competency  items 
Often  the  most  difficult  task  in  a 
organization’s  core  competencies,  and 
how  they  can  be  communicated  in  a 
survey.  According  to  Edwards  and 
Ewen  (1996),  the  best  way  to  select 
competencies  for  the  survey  is  to 
organize  a  focus  group  whose  job  is  to 
answer  the  question:  “  What  are  the 
critical  competencies  our  organization 
will  need  in  the  future  to  sustain  our 
competitive  advantage?”.  Within  the 
Army,  there  are  doctrinal  leadership 
competencies  which  can  be  used  as  a 
guide  to  develop  the  items  of  a  leadership  assessment  instrument.  Examining  established  doctrine 
is  an  excellent  place  to  start  when  determining  important  skills  and  behaviors  that  are  of  value  to 
an  organization.  As  a  general  rule,  competencies  should  be  written  clearly,  in  the  active  voice, 
and  be  brief.  Competencies  should  be  defined  with  illustrative  behaviors.  Recently,  the  Army 


multirater  feedback  project  is  to  identify  an 

•  Do  the  competencies  match  the  organization’s 

vision  and  values? 

•  Is  the  competency  set  complete? 

•  Is  there  a  focus  on  critical  behaviors  and  skills 

that  differentiate  the  organization  in  the 
marketplace? 

•  Is  the  organization’s  common  language  used? 

•  Is  simple  language  that  all  members  of  the 

organization  understand  used? 

•  Is  similar  content  grouped  together,  such  as 

competencies  associated  with  teamwork? 

•  Is  the  competency  set  simple  and  short? 


only  be  realized  if  a  number  of  critical 
Effective  Implementation  depends  on: 

•  A  relevant  assessment  instrument 

•  Involvement  of  target  audience 

•  Ensuring  confidentiality 

•  Training  for  participants 

•  Continuing  review  of  the  process 

«  Follow-up  support  for  self  development 
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Research  Institute  developed  the  Leader  Azimuth  Check  which  includes  items  based  directly  on 
new  leadership  doctrine.  For  instance,  the  new  doctrine  indicates  that  planning  and  organizing  is 
of  critical  importance  to  the  Army.  These  competencies  are  defined  with  statements  such  as: 
“develops  effective  plans  to  achieve  organizational  goals”,  “anticipates  how  different  plans  will 
look  when  executed”,  and  “sets  clear  priorities”. 

Finally,  considering  that  individuals  generally  receive  feedback  only  on  the  competencies 
specified  by  the  instrument,  a  multirater  assessment  instrument  can  determine  and  potentially 
restrict  the  feedback  received.  Even  in  cases  where  free-text  input  is  allowed,  written  evaluative 
comments  will  tend  to  address  the  competencies  contained  in  the  instrument.  For  example,  if 
there  are  no  items  concerning  an  individual’s  integrity,  then  no  feedback  on  this  topic  will  be 
generated  for  the  individual  receiving  feedback. 

2.  Involvement  of  the  target  audience 

In  order  to 

maximize  the  relevance  and 
utility  of  a  multirater 
instrument,  all  involved 
members  should  participate 
in  its  development.  Those 
being  evaluated  by  the  standards  they  set  may  have  especially  insightful  input.  Additionally,  the 
entire  360  process  should  involve  all  members  of  the  organization  including  top  executives.  The 
organization  should  rely  on  research-based  protocol  for  collecting  and  scoring  data.  The  selection 
teams  (i.e.,  the  group  of  people  providing  feedback)  should  consist  of  at  least  four  respondents  in 
addition  to  the  immediate  supervisor  and  self.  Finally,  an  understandable  process  that  is 
communicated  clearly  to  all  is  a  critical  feature  to  the  success  of  a  multirater  assessment  program. 

3.  Ensuring  confidentiality 

During  the  data  collection  process, 
anonymity  is  an  important  safeguard  ensuring 
accuracy  of  responses.  Anonymity  is  utilized  so 
that  the  source  of  an  individual’s  feedback  is  not 
identifiable  in  any  manner  by  that  individual.  If 
respondents  do  not  feel  confident  that  their  responses  will  remain  confidential,  they  may  provide 
uniformly  inflated  evaluations,  or  they  may  refuse  to  participate  at  all.  Results  from  a  follow  up 
survey,  after  a  multirater  feedback  procedure  was  implemented,  indicated  that  24%  of 
respondents  would  have  rated  their  bosses  differently  if  the  ratings  had  not  been  anonymous 
(London  et.  al.,  1990).  The  end  result  of  not  protecting  anonymity  is  that  those  who  provide  the 
feedback  do  not  trust  the  process,  and  those  who  receive  the  feedback  do  not  view  the  feedback 
as  credible  (Edwards  &  Ewen,  1996). 


•  A  source  outside  of  the  organization 

scores  the  assessment  and 
completes  the  feedback  report 

•  Feedback  is  provided  only  when 

there  is  more  than  one  respondent 


•  A  cross-functional  set  of  employees  are  surveyed 

•  Functional  users  review  the  survey 

•  The  survey  is  pretested  to  assess  readability, 

completeness,  and  clarity 

•  The  survey  is  refined  based  on  user  experience 
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4.  Training  for  participants 

Training  in  providing  and  receiving  feedback  is  critical  to  the  success  of  a  multirater 
feedback  program.  Besides  the 
obvious  value  of  having 
organizational  members  accurately 
completing  forms,  there  are  added 
benefits  of  training.  Training  can 
highlight  the  advantages  of  a  360, 
thereby  instilling  confidence  in  the 
process.  Moreover,  training  sets 
users  at  ease  with  the  process. 


•  An  introduction  to  the  concept  of  multirater  feedback  is 

provided  to  all  employees 

•  An  explanation  for  why  the  organization  is  adopting  the 

process  is  provided 

•  The  various  steps  and  safeguards  are  clarified 

•  Employees  are  trained  in  how  to  complete  the  feedback 

surveys,  how  to  interpret  their  results,  how  to  accept 
feedback  constructively  and  in  how  to  follow  up  with 
personal  plans  of  action _ 


5.  Continuing  review  of  process  and  instrument 

Research  indicates  that  the  process  for  conducting  a  360  degree  assessment  must  be 
formalized  to  ensure  fairness  and  accuracy.  Informal  systems  are  easy  and  fast  to  implement, 
however  they  fail  to  provide  sufficient  safeguards  to  ensure  fairness  and  are  more  likely  to 
multiply  rather  than  reduce  error  in  the  system.  (For  a  detailed  discussion  of  safeguards  that  can 
be  practiced  with  a  360,  see  Edwards  &  Ewen, 

1996).  The  dangers  of  an  informal  system 
involve  reported  false  information  that  may 
appear  credible  and  which  may  inappropriately 
impact  an  employee’s  self-perception  and/or 
career.  Furthermore,  an  informal  process  with 
its  associated  problems  may  negatively 

influence  one’s  willingness  to  participate  in  a  fair  and  formalized,  or  structured,  360  feedback 
system  at  a  later  date  (Edwards  &  Ewen,  1996). 


The  process  provides  sufficient  safeguards 
An  appeals  process  is  included  (especially  if 
the  360  is  used  for  performance  appraisal 
versus  developmental  purposes  only) 

The  process  is  formal 


6.  Follow-up 

Follow-up  is  a  critical  factor  in  the 
professional  development  of  members  of  an 
organization.  The  real  work  begins  when 
individuals  receive  their  feedback.  The 
multirater  feedback  process  can  provide 
individuals  with  the  information  they  need  to 
set  personal  goals  and  standards,  to  improve 
their  skills  and  keep  their  career  on  track.  Once  feedback  is  received  and  properly  interpreted,  the 
multirater  feedback  system  is  still  not  completed.  There  must  be  follow-through.  Follow-through 
means  more  than  creating  objectives  and  action  steps  based  on  feedback.  According  to  Kaplan 
(1993),  follow-through  should  be  viewed  as  a  project,  and  as  such,  it  should  be  managed  actively. 
There  are  various  methods  for  managing  the  process  and  thereby  boosting  the  effectiveness  of  the 
multirater  assessment  methodology. 


•  Individuals  are  provided  with  the 

information  needed  to  set  personal 
action  plans 

•  Individuals  are  encouraged/coached  as 

they  follow  through  with  their  personal 
action  plans 

•  Individuals  are  held  accountable  for 

executing  their  personal  action  plans 
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Enhancing  the  Effectiveness  of  a  Multirater  Feedback  System 

Organizations  wishing  to  boost  the  effectiveness  of  a  multirater  feedback  program  have  several 
options  which  may  include:  continuously  reviewing  the  feedback,  providing  feedback  in  the  form 
of  written  responses  to  open  ended  questions,  helping  recipients  of  feedback  develop  an 
understanding  of  what  is  motivating  their  behaviors,  and  examining  an  employee’s  personal  and 
career  history  (Kaplan,  1993). 

If  an  organization  decides  to  utilize  one  of  these  interventions,  caution  must  be  practiced  to 
protect  the  employee.  Organizations  must  choose  the  intervention  most  conducive  to  their 
climate  and  structure.  While  delving  into  an  employee’s  personal  and  career  history  may  seem  a 
viable  option  in  one  organization,  it  may  seem  intrusive  in  another. 

Impact  of  Multirater  Feedback  Programs 

Multirater  feedback  programs  are  based  on  the  assumptions  that  (a)  multirater  performance 
evaluations  provide  different  perspectives  from  which  to  detect  discrepancies  between  how  we 
view  ourselves  and  how  others  view  us,  (b)  awareness  of  discrepancies  enhances  self-awareness, 
and  (c)  enhanced  self-awareness  facilitates  performance.  However,  despite  the  growing 
popularity  of  multirater  feedback  programs,  there  is  very  little  research  testing  these  assumptions. 

In  one  study,  Hazucha,  Hezlett  &  Schneider  (1993)  attempted  to  evaluate  the  impact  of 
multirater  feedback  as  a  management  development  intervention.  They  had  managers  complete  a 
management  skills  profile  to  measure  skills  and  to  give  feedback.  Two  years  later,  they  conducted 
a  follow-up  with  48  of  those  managers.  They  reported  that  managers’  skills  increased  following 
multirater  feedback,  and  that  self-other  agreement  was  greater  two  years  after  the  initial  multirater 
process.  Additionally,  they  reported  that  management  skills  were  related  to  later  advancement. 
They  concluded  that  multirater  feedback  is  an  effective  management  development  tool.  Hazucha 
et.  al.  (1993)  also  administered  a  second  survey  and  found  that  the  development  activities  which 
were  most  strongly  related  to  skill  development  after  multirater  feedback  were  1)  reviewing  plans 
and  progress  at  least  quarterly,  2)  obtaining  input  into  development  plans  from  co-workers,  and  3) 
receiving  coaching  and  feedback. 

London  and  Wohlers  (1991)  conducted  an  examination  of  the  effects  of  an  upward  feedback 
intervention  on  profile  agreement  (which  is  the  agreement  between  self-ratings  and  averaged  other 
ratings).  Profile  agreement  was  fairly  low  across  the  sample,  however  it  improved  over  a  one- 
year  period  of  time.  These  results  indicate  that  feedback  from  others  raises  awareness  of 
strengths  and  weaknesses. 

While  these  two  studies  are  a  step  in  the  right  direction,  the  absence  of  a  control  or 
comparison  group  makes  it  difficult  to  determine  whether  the  performance  increases  occurred  as  a 
result  of  receiving  feedback.  An  additional  study  by  Smither,  London,  Vasilopoulos,  Reilly, 
Millsap,  and  Salvemini  (1995),  examined  the  impact  of  implementing  an  upward  feedback 
program  by  comparing  managers  who  received  individualized  feedback  to  managers  who  did  not. 
These  researchers  found  that,  based  on  subordinates’  ratings,  managers  whose  initial  level  of 
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performance  was  moderate  or  low  improved  their  performance  over  the  6-month  period  after 
having  received  feedback,  more  than  the  group  of  managers  who  received  no  individualized 
feedback.  This  (quasi-experimental)  study  was  limited  in  that  the  researchers  could  not  randomly 
assign  managers  to  the  two  conditions.  Thus,  while  multirater  feedback  programs  seem  promising, 
a  true  experimental  examination  of  the  impact  on  performance  is  still  warranted. 

Implications  for  Use  by  the  U.S.  Army 

The  Leader  Development  program  of  the  U.S.  Army  is  supported  by  the  three  pillars 
prescribed  by  doctrine:  institutional  training,  operational  assignments,  and  self-development. 

While  the  starting  place  for  any  self-development  plan  must  be  an  accurate  assessment  of  relevant 
strengths  and  weaknesses,  there  is  currently  little  to  guide  the  self-development  portion  of  this 
process.  Much  of  the  research  reviewed  in  this  paper  highlights  the  importance  of  others’ 
perspectives  in  forming  an  accurate  self-assessment.  The  multirater  feedback  process  may  be  the 
tool  the  Army  has  been  searching  for  to  create  permanent  changes  in  its  soldier's  performance 
through  increased  self-awareness  of  strengths  and  weaknesses.  However,  there  are  some  aspects 
of  multirater  feedback  systems  which  the  Army  needs  to  further  develop. 

In  general,  if  the  Army  wants  a  better  instrument  to  work  with,  more  investigation  of  the 
psychometric  properties  that  work  best  with  a  multirater  survey  is  necessary.  In  particular,  are 
wider  scales  necessary  so  that  feedback  appears  in  a  manner  that  is  more  motivating?  Do 
behavioral  anchors  on  survey  questions  add  to  the  quality  of  feedback  and  interpretability?  How 
can  we  balance  the  need  to  thoroughly  cover  all  competencies  with  the  need  to  keep  the  survey 
length  reasonable,  especially  when  individuals  may  be  completing  surveys  for  more  than  one 
individual?  How  should  feedback  be  graphically  displayed  so  that  it  is  most  easily  interpreted? 

Future  research  efforts  should  attempt  to  determine  how  the  organizational  structure  of  the 
Army  (compared  to  corporate  structures)  influences  the  utilization  and  acceptance  of  feedback. 
How  can  individuals  in  the  Army  be  supported  during  the  follow-through  period  of  remediation? 
Strategies  and  interventions,  which  could  enhance  the  effectiveness  of  the  360  process,  should  be 
developed  to  meet  the  specific  needs  of  the  Army. 

Finally,  it  would  be  beneficial  to  determine  empirically  whether  multirater  assessments  create 
permanent  increases  in  self-awareness  and  self-development  which  could  contribute  substantially 
to  meeting  the  long  term  goals  of  an  organization  such  as  the  Army. 
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